DAOS-18609 test: Resolving slurm issue for el9 by saurabhtandan · Pull Request #18294 · daos-stack/daos

saurabhtandan · 2026-05-19T20:55:01Z

After CI moved over to el9 slurm package
installation needed readjusting.
slurm-example-configs removed
slurm-slurmdbd added

Quick-build: true
Quick-Functional: true
Test-tag: test_soak_smoke

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

After CI moved over to el9 slurm package installation needed readjusting. slurm-example-configs removed slurm-slurmdbd added Quick-build: true Quick-Functional: true Test-tag: test_soak_smoke Signed-off-by: Saurabh Tandan <saurabh.tandan@hpe.com>

github-actions · 2026-05-19T20:55:59Z

Ticket title is 'launch/functional_hardware_large - Error setting up slurm - Unable to find a match: slurm-example-configs'
Status is 'In Progress'
Labels: '2.6.5.p1rc1,2.6.5.p2rc1,2.6.5rc1,2.6.5rc2,2.6.5rc3,2.8.0rc2,2.8.0rc3,2.8.0rc4,2.8.0rc5,ci_master_weekly,testp1,weekly_test'
https://daosio.atlassian.net/browse/DAOS-18609

daosbuild3 · 2026-05-19T22:52:40Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18294/1/execution/node/742/log

Quick-build: true Quick-Functional: true Test-tag: test_soak_smoke Signed-off-by: Saurabh Tandan <saurabh.tandan@hpe.com>

daosbuild3 · 2026-06-08T23:48:53Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18294/2/execution/node/742/log

Quick-build: true Quick-Functional: true Test-tag: test_soak_smoke Signed-off-by: Saurabh Tandan <saurabh.tandan@hpe.com>

daosbuild3 · 2026-06-09T20:58:36Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18294/3/execution/node/742/log

Quick-build: true Quick-Functional: true Test-tag: test_soak_smoke Signed-off-by: Saurabh Tandan <saurabh.tandan@hpe.com>

daosbuild3 · 2026-07-06T13:56:46Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18294/5/execution/node/779/log

daosbuild3 · 2026-07-27T16:48:56Z

Test stage Test RPMs on EL 9.6 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18294/6/execution/node/558/log

Test-tag: test_soak_smoke

daosbuild3 · 2026-08-19T04:10:23Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18294/8/execution/node/1029/log

is not being picked for el9 Test-tag: test_soak_smoke Signed-off-by: Saurabh Tandan <saurabh.tandan@hpe.com>

Test-tag: test_soak_smoke Signed-off-by: Saurabh Tandan <saurabh.tandan@hpe.com>

daosbuild3 · 2026-08-20T04:26:46Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18294/9/execution/node/1028/log

phender

The majority of these changes don't seem to affect fixing the slurm RPM install issue:

2026/08/20 04:26:23 INFO                install_packages: Installing packages on hdr-[140,142-149]: slurm, slurm-example-configs, slurm-slurmctld, slurm-slurmd
2026/08/20 04:26:23 DEBUG                     run_remote: Running on hdr-[140,142-149] with a 600 second timeout: sudo -n dnf install -y slurm slurm-example-configs slurm-slurmctld slurm-slurmd
2026/08/20 04:26:26 DEBUG                log_result_data:   hdr-144 (rc=1):
2026/08/20 04:26:26 DEBUG                log_result_data:     created by dnf config-manager from https://arti  82 kB/s | 3.0 kB     00:00    
2026/08/20 04:26:26 DEBUG                log_result_data:     daos_ci-rocky9-base-artifactory                 5.1 kB/s | 2.0 kB     00:00    
2026/08/20 04:26:26 DEBUG                log_result_data:     daos_ci-rocky9-extras-artifactory               2.7 kB/s | 1.5 kB     00:00    
2026/08/20 04:26:26 DEBUG                log_result_data:     daos_ci-rocky9-appstream-artifactory             15 kB/s | 2.4 kB     00:00    
2026/08/20 04:26:26 DEBUG                log_result_data:     daos_ci-rocky9-crb-artifactory                  7.6 kB/s | 2.4 kB     00:00    
2026/08/20 04:26:26 DEBUG                log_result_data:     daos_ci-el9-epel-artifactory                     93 kB/s | 4.0 kB     00:00    
2026/08/20 04:26:26 DEBUG                log_result_data:     daos-stack-deps-el-9-stable-local-artifactory   5.2 kB/s | 478  B     00:00    
2026/08/20 04:26:26 DEBUG                log_result_data:     daos_ci-intel-tools-el-8-stable-local-artifacto 5.1 kB/s | 478  B     00:00    
2026/08/20 04:26:26 DEBUG                log_result_data:     created by dnf config-manager from https://jenk  77 kB/s | 3.0 kB     00:00    
2026/08/20 04:26:26 DEBUG                log_result_data:     No match for argument: slurm-example-configs
2026/08/20 04:26:26 DEBUG                log_result_data:     Error: Unable to find a match: slurm-example-configs

Have you tried removing slurm-example-configs from https://github.com/daos-stack/daos/blob/master/src/tests/ftest/slurm_setup.py#L37

phender · 2026-08-20T13:25:11Z

The copyright needs updating - check your githooks

Suggested change

(C) Copyright 2025-2026 Hewlett Packard Enterprise Development LP

phender · 2026-08-20T13:25:15Z

    return commands


 def create_app_cmdline(self, job_spec, pool, ppn, nodesperjob):


We should add a `# pylint: disable=too-many-locals

phender · 2026-08-20T13:26:52Z


        # Restart slurmd on all nodes
-        self._restart_systemctl(self.all_nodes, 'slurmd', '/var/log/slurmd.log', self.SLURM_CONF)
+        self._restart_systemctl(


Why does this need to be modified? The original line length does not exceed the limit.

phender · 2026-08-20T13:27:28Z

        self._modify_slurm_config_file(
-            'epilog file', self.all_nodes, 's#EpilogSlurmctld=#EpilogSlurmctld={EPILOG_FILE}#g',
-            self.root)
+            'epilog file', self.all_nodes,


Same question

phender · 2026-08-20T13:28:03Z

            if "Socket" in info and "Core" in info and "Thread" in info:
                echo_command = (f'echo \"Nodename={nodelist} Sockets={info["Socket"]} '
-                                f'CoresPerSocket={info["Core"]} ThreadsPerCore={info["Thread"]}\"')
+                                f'CoresPerSocket={info["Core"]} ThreadsPerCore={info["Thread"]}\" ')


Is this extra space needed?

phender · 2026-08-20T13:29:42Z

+    os_name = (os_info.name or "").lower()
+    os_version = str(os_info.version or "").strip()
+
+    # Fallback for cases where avocado detect() returns unknown/0.


Where are we running into cases where avocado detect() returns unknown/0? We use detect() in every CI test in https://github.com/daos-stack/daos/blob/master/src/tests/ftest/util/apricot/apricot/test.py#L156 and it works with all of the distros we run on.

Note: Discard results for Build#9 as wrong slurm_setup.py was accidentally pushed.

Regarding detect(), as noticed for the results of build#8, it was not able to detect the correct os version for el9 hence was not able to copy the correct lammps build hence lammps jobs failed. So I am just trying to bullet proof that in scenarios where it is failing to detect is especially in case of el9. Not sure what exactly is detect() used for in other tests, but here I need the version number as well.

phender · 2026-08-20T13:36:41Z

+        os_name = (os_release.get("NAME") or os_release.get("ID") or os_name).lower()
+        os_version = (os_release.get("VERSION_ID") or os_version).strip()
+
+    if os.environ.get("DAOS_TEST_MODE") is None:


This may be a bit out of scope for this PR, but it seems all DAOS_TEST_MODE does is prevent the code from updating DAOS_TEST_APP_DIR and I_MPI_OFI_LIBRARY_INTERNAL in certain situations. It seems like the updating of those variables should be dependent upon whether or not they are already set - and then we don't need DAOS_TEST_MODE.

Test-tag: test_soak_smoke Signed-off-by: Saurabh Tandan <saurabh.tandan@hpe.com>

saurabhtandan · 2026-08-20T16:36:11Z

The majority of these changes don't seem to affect fixing the slurm RPM install issue:

2026/08/20 04:26:23 INFO                install_packages: Installing packages on hdr-[140,142-149]: slurm, slurm-example-configs, slurm-slurmctld, slurm-slurmd
2026/08/20 04:26:23 DEBUG                     run_remote: Running on hdr-[140,142-149] with a 600 second timeout: sudo -n dnf install -y slurm slurm-example-configs slurm-slurmctld slurm-slurmd
2026/08/20 04:26:26 DEBUG                log_result_data:   hdr-144 (rc=1):
2026/08/20 04:26:26 DEBUG                log_result_data:     created by dnf config-manager from https://arti  82 kB/s | 3.0 kB     00:00    
2026/08/20 04:26:26 DEBUG                log_result_data:     daos_ci-rocky9-base-artifactory                 5.1 kB/s | 2.0 kB     00:00    
2026/08/20 04:26:26 DEBUG                log_result_data:     daos_ci-rocky9-extras-artifactory               2.7 kB/s | 1.5 kB     00:00    
2026/08/20 04:26:26 DEBUG                log_result_data:     daos_ci-rocky9-appstream-artifactory             15 kB/s | 2.4 kB     00:00    
2026/08/20 04:26:26 DEBUG                log_result_data:     daos_ci-rocky9-crb-artifactory                  7.6 kB/s | 2.4 kB     00:00    
2026/08/20 04:26:26 DEBUG                log_result_data:     daos_ci-el9-epel-artifactory                     93 kB/s | 4.0 kB     00:00    
2026/08/20 04:26:26 DEBUG                log_result_data:     daos-stack-deps-el-9-stable-local-artifactory   5.2 kB/s | 478  B     00:00    
2026/08/20 04:26:26 DEBUG                log_result_data:     daos_ci-intel-tools-el-8-stable-local-artifacto 5.1 kB/s | 478  B     00:00    
2026/08/20 04:26:26 DEBUG                log_result_data:     created by dnf config-manager from https://jenk  77 kB/s | 3.0 kB     00:00    
2026/08/20 04:26:26 DEBUG                log_result_data:     No match for argument: slurm-example-configs
2026/08/20 04:26:26 DEBUG                log_result_data:     Error: Unable to find a match: slurm-example-configs

Have you tried removing slurm-example-configs from https://github.com/daos-stack/daos/blob/master/src/tests/ftest/slurm_setup.py#L37

Yeah I did in the previous commits. Along with removal of slurm-example-configs we need to add

The majority of these changes don't seem to affect fixing the slurm RPM install issue:

2026/08/20 04:26:23 INFO                install_packages: Installing packages on hdr-[140,142-149]: slurm, slurm-example-configs, slurm-slurmctld, slurm-slurmd
2026/08/20 04:26:23 DEBUG                     run_remote: Running on hdr-[140,142-149] with a 600 second timeout: sudo -n dnf install -y slurm slurm-example-configs slurm-slurmctld slurm-slurmd
2026/08/20 04:26:26 DEBUG                log_result_data:   hdr-144 (rc=1):
2026/08/20 04:26:26 DEBUG                log_result_data:     created by dnf config-manager from https://arti  82 kB/s | 3.0 kB     00:00    
2026/08/20 04:26:26 DEBUG                log_result_data:     daos_ci-rocky9-base-artifactory                 5.1 kB/s | 2.0 kB     00:00    
2026/08/20 04:26:26 DEBUG                log_result_data:     daos_ci-rocky9-extras-artifactory               2.7 kB/s | 1.5 kB     00:00    
2026/08/20 04:26:26 DEBUG                log_result_data:     daos_ci-rocky9-appstream-artifactory             15 kB/s | 2.4 kB     00:00    
2026/08/20 04:26:26 DEBUG                log_result_data:     daos_ci-rocky9-crb-artifactory                  7.6 kB/s | 2.4 kB     00:00    
2026/08/20 04:26:26 DEBUG                log_result_data:     daos_ci-el9-epel-artifactory                     93 kB/s | 4.0 kB     00:00    
2026/08/20 04:26:26 DEBUG                log_result_data:     daos-stack-deps-el-9-stable-local-artifactory   5.2 kB/s | 478  B     00:00    
2026/08/20 04:26:26 DEBUG                log_result_data:     daos_ci-intel-tools-el-8-stable-local-artifacto 5.1 kB/s | 478  B     00:00    
2026/08/20 04:26:26 DEBUG                log_result_data:     created by dnf config-manager from https://jenk  77 kB/s | 3.0 kB     00:00    
2026/08/20 04:26:26 DEBUG                log_result_data:     No match for argument: slurm-example-configs
2026/08/20 04:26:26 DEBUG                log_result_data:     Error: Unable to find a match: slurm-example-configs

Have you tried removing slurm-example-configs from https://github.com/daos-stack/daos/blob/master/src/tests/ftest/slurm_setup.py#L37

Yeah I did, along with removal slurm-example-configs of we need 'slurm-slurmdbd' for it to work on el9. Unfortunately, I pushed the wrong version of this file in the last commit. Previous commits had the correct version of file and the soak smoke have even run successfully for Jenkins build #8 for this PR. But there were a few lammps jobs which failed because it was not able to find the lammps binaries. That's the portion I am working on now.

Test-tag: test_soak_smoke Signed-off-by: Saurabh Tandan <saurabh.tandan@hpe.com>

Corrected the log file path

3f6561e

Quick-build: true Quick-Functional: true Test-tag: test_soak_smoke Signed-off-by: Saurabh Tandan <saurabh.tandan@hpe.com>

Modified slurm_setup.py file

eabcdb9

Quick-build: true Quick-Functional: true Test-tag: test_soak_smoke Signed-off-by: Saurabh Tandan <saurabh.tandan@hpe.com>

saurabhtandan added 2 commits July 2, 2026 17:27

Modified utils/slurm_utils.py

b3b45ca

Quick-build: true Quick-Functional: true Test-tag: test_soak_smoke Signed-off-by: Saurabh Tandan <saurabh.tandan@hpe.com>

Modified utils/slurm_utils.py

8ca40a7

Quick-build: true Quick-Functional: true Test-tag: test_soak_smoke Signed-off-by: Saurabh Tandan <saurabh.tandan@hpe.com>

Merge branch 'master' into standan/DAOS-18609

281778b

Test-tag: test_soak_smoke

saurabhtandan force-pushed the standan/DAOS-18609 branch from a17096f to 281778b Compare August 18, 2026 20:10

saurabhtandan added 3 commits August 19, 2026 10:44

Merge branch 'master' into standan/DAOS-18609

c488fa9

Updated os parsing code as it correct os version

61d79c0

is not being picked for el9 Test-tag: test_soak_smoke Signed-off-by: Saurabh Tandan <saurabh.tandan@hpe.com>

Disabling vpic until the build is ready.

5227ca5

Test-tag: test_soak_smoke Signed-off-by: Saurabh Tandan <saurabh.tandan@hpe.com>

phender reviewed Aug 20, 2026

View reviewed changes

saurabhtandan added 2 commits August 20, 2026 15:30

Re-correct slurm_setup.py

2d7e809

Test-tag: test_soak_smoke Signed-off-by: Saurabh Tandan <saurabh.tandan@hpe.com>

Resolve linting issues

6bec61f

Test-tag: test_soak_smoke Signed-off-by: Saurabh Tandan <saurabh.tandan@hpe.com>

Correct liniting error

5918daa

Test-tag: test_soak_smoke Signed-off-by: Saurabh Tandan <saurabh.tandan@hpe.com>


	(C) Copyright 2025-2026 Hewlett Packard Enterprise Development LP

		return commands


		def create_app_cmdline(self, job_spec, pool, ppn, nodesperjob):

Uh oh!

Conversation

saurabhtandan commented May 19, 2026

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions Bot commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild3 commented May 19, 2026

Uh oh!

daosbuild3 commented Jun 8, 2026

Uh oh!

daosbuild3 commented Jun 9, 2026

Uh oh!

daosbuild3 commented Jul 6, 2026

Uh oh!

daosbuild3 commented Jul 27, 2026

Uh oh!

daosbuild3 commented Aug 19, 2026

Uh oh!

daosbuild3 commented Aug 20, 2026

Uh oh!

phender left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

saurabhtandan commented Aug 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented May 19, 2026 •

edited

Loading